Near-eye display with extended accommodation range adjustment

ABSTRACT

A near-eye display system includes a display panel to display a near-eye lightfield frame comprising an array of elemental images and an eye tracking component to track a pose of a user&#39;s eye. The system further includes a lenslet array and a rendering component to adjust the focal points of the array of elemental images in the integral lightfield frame based on the pose of the user&#39;s eye. A method of operation of the near-eye display system includes determining, using an eye tracking component of the near-eye display system, a first pose of a user&#39;s eye and determining a desired focal point for an array of elemental images forming an integral lightfield frame based on the first pose of the user&#39;s eye. The method further includes changing the focal length of light projecting out of a lenslet array based on the first pose of the user&#39;s eye.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. Provisional Patent Application62/511,567, entitled “NEAR-EYE DISPLAY WITH EXTENDED ACCOMMODATION RANGEADJUSTMENT” and filed on May 26, 2017, the entirety of which isincorporated by reference herein.

BACKGROUND

Head-mounted displays (HMDs) and other near-eye display systems canutilize an integral lightfield display or other computational display toprovide effective display of three-dimensional (3D) graphics. Generally,the integral lightfield display employs one or more display panels andan array of lenslets, pinholes, or other optic features that overlie theone or more display panels. A rendering system renders an array ofelemental images, with each elemental image representing an image orview of an object or scene from a corresponding perspective or virtualcamera position. Such integral lightfield displays typically exhibit atradeoff between resolution and accommodation range as resolution isproportional to the density of lenslets. Thus, to provide satisfactoryresolution, a conventional near-eye display system employing an integrallightfield display typically has a low density of large-sized lenslets,which limits the display resolution or has a high density ofsmaller-sized lenslets that limit the accommodation range.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure may be better understood, and its numerousfeatures and advantages made apparent to those skilled in the art byreferencing the accompanying drawings. The use of the same referencesymbols in different drawings indicates similar or identical items.

FIG. 1 is a diagram illustrating a near-eye display system employing eyetracking and corresponding elemental image shifting to provide dynamicfocal length adjustment in accordance with some embodiments.

FIG. 2 is a diagram illustrating an example of dynamic focal lengthadjustment in the near-eye display system of FIG. 1 in accordance withsome embodiments.

FIG. 3 is a diagram illustrating an additional example of dynamic focallength adjustment in the near-eye display system of FIG. 1 in accordancewith some embodiments.

FIG. 4 is a flow diagram illustrating a method for dynamic focal lengthadjustment in the near-eye display system of FIG. 1 in accordance withsome embodiments.

FIG. 5 is a diagram illustrating an example of dynamic accommodationrange adjustment in the near-eye display system of FIG. 1 in accordancewith some embodiments.

FIG. 6 is a diagram illustrating an example varifocal lenslet array fordynamic focal length adjustment in the near-eye display system of FIG. 1in accordance with some embodiments.

DETAILED DESCRIPTION

FIGS. 1-6 illustrate example methods and systems for dynamic focallength and accommodation range adjustment based on user eye pose in anear-eye display system. In at least one embodiment, the near-eyedisplay system employs a computational display to display integrallightfield frames of imagery to a user so as to provide the user with animmersive virtual reality (VR) or augmented reality (AR) experience.Each integral lightfield frame is composed of an array of elementalimages, with each elemental image representing a view of an object orscene from a different corresponding viewpoint. An array of lensletsoverlies the display panel and operates to present the array ofelemental images to the user as a single autostereoscopic image.

As the resolution of computational display are proportional to the ratioof lenslet size to lenslet focal length, an attempt to increaseresolution using large lenslets generally results in reduced focallengths and accommodation ranges, and vice versa. To provide improvedresolution without a corresponding reduction in accommodation range, inat least one embodiment the near-eye display systems described hereinutilize a dynamic technique wherein an eye tracking component isutilized to determine the current pose (position and/or rotation) of theuser's eye and, based on this current pose, determine a voltage to beapplied to a variable-index material by which light projected out oflenslets have their focal lengths changed so as to change how in focusportions of an image are perceived based on the current pose of theuser's eye. As an example, the refractive index of the material mayinitially be set to generate a first accommodation range within whichobjects may be perceived in focus. Subsequently, the refractive index ofthe material may be changed to generate a second accommodation rangewithin which objects may be perceived in focus. As the user's gazechanges, the refractive index of the material is changed to dynamicallyadjust the accommodation range within which objects may be perceived infocus. Thus, dynamically changing the refractive index and shifting theaccommodation range responsive to changes in the pose of the eye in theuser effectively provides for a large accommodation range withoutrequiring a corresponding reduction in the resolution of the near-eyedisplay system.

FIG. 1 illustrates a near-eye display system 100 incorporating dynamicaccommodation range adjustment in accordance with at least oneembodiment. In the depicted example, the near-eye display system 100includes a computational display sub-system 102, a rendering component104, and one or more eye tracking components, such as one or both of aneye tracking component 106 for tracking a user's left eye and an eyetracking component 108 for tracking the user's right eye. Thecomputational display sub-system 102 includes a left-eye display 110 anda right-eye display 112 mounted in an apparatus 114 (e.g., goggles,glasses, etc.) that places the displays 110, 112 in front of the leftand right eyes, respectively, of the user.

As shown by view 116, each of the displays 110, 112 includes at leastone display panel 118 to display a sequence or succession of integrallightfield frames (hereinafter, “lightfield frame” for ease ofreference), each of which comprises an array 120 of elemental images122. For ease of reference, an array 120 of elemental images 122 mayalso be referred to herein as a lightfield frame 120. Each of thedisplays 110, 112 further includes an array 124 of lenslets 126 (alsocommonly referred to as “microlenses”) overlying the display panel 118.Typically, the number of lenslets 126 in the lenslet array 124 is equalto the number of elemental images 122 in the array 120, but in otherimplementations the number of lenslets 126 may be fewer or greater thanthe number of elemental images 122. Note that while the example of FIG.1 illustrates a 5×4 array of elemental images 122 and a corresponding5×4 array 120 of lenslets 126 for ease of illustration, in a typicalimplementation the number of elemental images 122 in a lightfield frame120 and the number of lenslets 126 in the lenslet array 124 typically ismuch higher. Further, in some embodiments, a separate display panel 118is implemented for each of the displays 110, 112, whereas in otherembodiments the left-eye display 110 and the right-eye display 112 sharea single display panel 118, with the left half of the display panel 118used for the left-eye display 110 and the right half of the displaypanel 118 used for the right-eye display 112.

Cross-view 128 of FIG. 1 depicts a cross-section view along line A-A ofthe lenslet array 124 overlying the display panel 118 such that thelenslet array 124 overlies the display surface 130 of the display panel118 so as to be disposed between the display surface 130 and thecorresponding eye 132 of the user. In this configuration, each lenslet126 focuses a corresponding region of the display surface 130 onto thepupil 134 of the eye, with each such region at least partiallyoverlapping with one or more adjacent regions. Thus, in suchcomputational display configurations, when an array 120 of elementalimages 122 is displayed at the display surface 130 of the display panel118 and then viewed by the eye 132 through the lenslet array 124, theuser perceives the array 120 of elemental images 122 as a single imageof a scene. Thus, when this process is performed in parallel for boththe left eye and right eye of the user with the proper parallaximplemented therebetween, the result is the presentation ofautostereoscopic three-dimensional (3D) imagery to the user.

As also shown in FIG. 1, the rendering component 104 includes a set ofone or more processors, such as the illustrated central processing unit(CPU) 136 and graphics processing units (GPUs) 138, 140 and one or morestorage components, such as system memory 142, to store softwareprograms or other executable instructions that are accessed and executedby the processors 136, 138, 140 so as to manipulate the one or more ofthe processors 136, 138, 140 to perform various tasks as describedherein. Such software programs include, for example, rendering program144 comprising executable instructions for an accommodation rangeadjustment process, as described below, as well as an eye trackingprogram 146 comprising executable instructions for an eye trackingprocess, as also described below.

In operation, the rendering component 104 receives rendering information148 from a local or remote content source 150, where the renderinginformation 148 represents graphics data, video data, or other datarepresentative of an object or scene that is the subject of imagery tobe rendered and displayed at the display sub-system 102. Executing therendering program 144, the CPU 136 uses the rendering information 148 tosend drawing instructions to the GPUs 138, 140, which in turn utilizethe drawing instructions to render, in parallel, a series of lightfieldframes 151 for display at the left-eye display 110 and a series oflightfield frames 153 for display at the right-eye display 112 using anyof a variety of well-known VR/AR computational/lightfield renderingprocesses. As part of this rendering process, the CPU 136 may receivepose information 150 from an inertial management unit (IMU) 154, wherebythe pose information 150 is representative of a current pose of thedisplay sub-system 102 and control the rendering of one or more pairs oflightfield frames 151, 153 to reflect the viewpoint of the object orscene from the current pose.

As described in detail below, the rendering component 104 further mayuse eye pose information from one or both of the eye tracking components106, 108 to shift the focal length of projections of elemental images122 from the lenslet array 124 to the eye 132 for the lightfield frameto be displayed, and thereby adjusting the focus of one or more of theelemental images 122 for the lightfield frame so displayed. To this end,the eye tracking components 106, 108 each may include one or moreinfrared (IR) light sources (referred to herein as “IR illuminators) toilluminate the corresponding eye with IR light, one or more imagingcameras to capture the IR light reflected off of the corresponding eyeas a corresponding eye image (eye image information 156), one or moremirrors, waveguides, beam splitters, and the like, to direct thereflected IR light to the imaging cameras, and one or more processors toexecute the eye tracking program 146 so as to determine a currentposition, current orientation, or both (singularly or collectivelyreferred to herein as “pose”) of the corresponding eye from the capturedeye image. Any of a variety of well-known eye tracking apparatuses andtechniques may be employed as the eye tracking components 146, 148 totrack one or both eyes of the user.

In a conventional computational display-based system, the properties ofthe lenslet array overlaying a display are typically fixed (that is, thephysical dimensions and/or the material construction of the lenslets arefixed, and often are the same for all the lenslets), which in turnresults in the optical properties of the lenslets being fixed. As aresult, changing the focus at which the user perceives the displayedimagery often includes mechanical actuation to physically move thelenslet array closer to or further away from the user's eyes. Innear-eye display systems, the small focal length of lenslets subjectsthem to small lens-display spacing tolerances. As a result, anyinaccuracies in the initial construction of the lenslet array orinaccuracies in mechanical translation during operation can result inunintended impacts to the user's perception of the displayed imagery,such as loss of focus or blurry objects in displayed imagery.

As described herein, in at least one embodiment the near-eye displaysystem 100 improves the accuracy of adjustments to the focus ofdisplayed imagery by implementing a variable-focal-length lenslet arrayconfigured to adjust the focal length of projected imagery to moreclosely align with the current pose of the user's eye. This isaccomplished by using the eye tracking components 106, 108 to track oneor both eyes of the user so as to determine the current pose of one orboth of the eyes for a corresponding lightfield frame to be displayed.With the current pose determined, the rendering component 104 thenelectrically adjusts the focal length of light projected from one ormore of the lenslets 126 in the lenslet array to change the focus pointof one or more of the elemental images 122 within a lightfield framebeing rendered relative to the user's eye 132. This change to the focuspoint brings objects as displayed by the display panel 118 into- orout-of-focus from the user's perspective. In this manner, the focallength(s) of the lenslets 126 may be dynamically adjusted to betteraccommodate the current pose of the user's eye.

In some embodiments, the lenslet array 124 includes lenslets constructedfrom nematic liquid crystal cells. The nematic liquid crystal cells areelectrically addressable using, for example, a voltage source (notshown). Changes in an applied voltage to the lenslets 126 cause therefractive index of the lenslet to change, thereby changing the focallength of the lenslets. In other embodiments, rather than using lensletsconstructed from nematic liquid crystal cells, a layer of variable-indexmaterial 158 (such as constructed out of nematic liquid crystal cells orother variable-focus optical components configured to have variablefocal lengths) is positioned so as to be disposed between the displaypanel 118 and the lenslet array 124.

Although described here in the context of nematic liquid crystals, thoseskilled in the art will recognize that any variable-index materialand/or variable-focus optical component may be used without departingfrom the scope of this disclosure. For example, such optical componentscan include, but is not limited to, deformable membrane mirrors (DMMs),fluid lenses, spatial light modulators (SLMs), electro-optical polymers,etc. Additionally, in some other embodiments, focal length of lightprojected from the lenslet array 124 may further be adjusted bycombining the variable-index lenslets or layer of variable-indexmaterial with a mechanical actuator (not shown) to change the physicaldistance between the lenslet array 124, the layer of variable-indexmaterial 158, the display panel 118, and the eye 132. For example, suchmechanical actuators may include piezo-electric, voice-coil, or electroactive polymer actuators.

In one embodiment, a voltage is applied to the lenslet array 124 or thelayer of variable-index material 158 as a whole. Accordingly, eachindividual lenslet 126 or the entire layer of variable-index material158 receives the same voltage for adjusting its refractive index,thereby changing the focal length of light projected from the entirelenslet array 124. This achieves the same effect as mechanicallyactuating and translating the lenslet array 124 closer to or furtheraway from the eye 132, and further improves the accuracy of achievingthe desired focal length. In another embodiment, each of the lenslets126 are individually addressable and can receive a different voltagefrom one another. Similarly, the layer of variable-index material 158may be pixelated with dimensions matching that of the lenslet array;each of the pixelated areas of the layer of variable-index material 158may be individually addressable. This allows greater control over thefocal length of light projected from each lenslet 126. Accordingly, thefocal length of light projected from each lenslet 126 is modulable andeach of the elemental images 122 may represent different portions of animage with a different viewing distance to objects in the image. Toprovide further granularity in control over focal points, in someembodiments, the layer of variable-index material 158 may be pixelatedat a sub-lenslet level with dimensions such that different portions ofeach elemental image 122 corresponding to each lenslet 126 may beindividually addressed to a unique focal length.

Alternatively, to provide further granularity in control over focalpoints, in other embodiments, the near-eye display system 100 includesan optional phase mask 160 positioned so as to be disposed between thedisplay panel 118 and the lenslet array 124. For example, as illustratedin FIG. 1, the optional phase mask 160 is a pixelated spatial lightmodulator (SLM) that receives incoming light from the display 118 (or insome embodiments, from variable-index material 158) and spatiallymodulates the phase of the output light beam. Accordingly, each lenslet126 would receive an incident beam having a plurality of spatiallyvarying phases for different rays such that different portions of eachelemental image 122 may be focused to at different focal lengths.

In other embodiments, the lenslet array 124 or the layer ofvariable-index material 158 may be segmented into two or morepartitions. Each partition may be addressed with the same voltage,thereby changing the focal length for that portion only. For example,the lenslet array 124 or the layer of variable-index material 158 may besegmented into four equal quadrants that each receive a differentvoltage signal, into individually addressable rows, into individuallycolumns, etc. Those skilled in the art will recognize that anysegmentation of the lenslet array 124 or the layer of variable-indexmaterial 158 into spatially-varying addressable partitions may be usedwithout departing from the scope of this disclosure.

To illustrate, FIG. 2 depicts a cross-section view 200 of acomputational display such as the ones utilized in the near-eye displaysystem 100 using variable-index lenslets. As shown in this view, each ofthe lenslets 126 of the lenslet array 124 serves as a separate“projector” onto the eye 132, with each “projector” overlapping with oneor more adjacent projectors in forming a composite virtual image 202from the array of elemental images displayed at the display panel 118.To illustrate, the lenslet 126-1 projects a corresponding elementalimage (represented by region 204) from region 210 of the virtual image202, the lenslet 126-2 projects a corresponding elemental image(represented by region 206) from region 212 of the virtual image 202,and the lenslet 126-3 projects a corresponding elemental image(represented by region 208) from region 214 of the virtual image 202. Asshown by FIG. 2, regions 210 and 212 overlap in sub-region 216, regions212 and 214 overlap in sub-region 220, and all three regions 210, 212,214 overlap in sub-region 218.

Thus, assuming in this example that an elemental image positioned atregion 206 of the display panel 118 is focused on by the eye 132 at afirst time t₁, the refractive index of lenslet 126-2 may be computed(e.g., by rendering component 104) and electrically changed such thatlight containing image data from lenslet 126-2 is focused at a firstfocal point 222 at the back of the eye 132. Accordingly, the region 212portion of virtual image 202 would appear to be in focus at the firsttime t₁. Subsequently, assuming in this example that the user looks awayat a second time t₂ to focus on an elemental image positioned at region204 of the display panel 118. To account for the change to a new posefor the user's eye, the refractive index of lenslet 126-2 may becomputed (e.g., by rendering component 104) and electrically changedsuch that light containing image data from lenslet 126-2 is focused at asecond focal point 224 such that the accommodation range of the user'seyes cannot bring such that portion of the image into focus.Accordingly, the region 212 portion of virtual image 202 would appear tobe out of focus (e.g., blurry) at the second time t₂.

In an alternative embodiment, FIG. 3 depicts a cross-section view 300 ofa computational display such as the ones utilized in the near-eyedisplay system 100 using a layer of variable-index material. As shown inthis view, each of the lenslets 126 of the lenslet array 124 serves as aseparate “projector” onto the eye 132, with each “projector” overlappingwith one or more adjacent projectors in forming a composite virtualimage 202 from the array of elemental images displayed at the displaypanel 118. To illustrate, the lenslet 126-1 projects a correspondingelemental image (represented by region 304) from region 310 of thevirtual image 302, the lenslet 126-2 projects a corresponding elementalimage (represented by region 306) from region 312 of the virtual image302, and the lenslet 126-3 projects a corresponding elemental image(represented by region 308) from region 314 of the virtual image 302. Asshown by FIG. 3, regions 310 and 312 overlap in sub-region 316, regions312 and 314 overlap in sub-region 320, and all three regions 310, 312,314 overlap in sub-region 318. In the embodiment illustrated in FIG. 3,the layer of variable-index material 158 (such as previously discussedwith respect to FIG. 1) changes its refractive index to change theincidence of light on the lenslets 126, which in turn changes the focaldistance of light projected from the lenslets 126.

Thus, assuming in this example that an elemental image positioned atregion 306 of the display panel 118 is focused on by the eye 132 at afirst time t₁, the refractive index of the layer of variable-indexmaterial 158 may be computed (e.g., by rendering component 104) andelectrically changed such that light containing image data projectedfrom lenslet 126-2 is focused at a first focal point 322 at the back ofthe eye 132. Accordingly, the region 312 portion of virtual image 302would appear to be in focus at the first time t₁. Subsequently, assumingin this example that the user looks away at a second time t₂ to focus onan elemental image positioned at region 304 of the display panel 118. Toaccount for the change to a new pose for the user's eye, the refractiveindex of the layer of variable-index material 158 may be computed (e.g.,by rendering component 104) and electrically changed such that lightcontaining image data from lenslet 126-2 is focused at a second focalpoint 324 such that the accommodation range of the user's eyes cannotbring such that portion of the image into focus. Accordingly, the region312 portion of virtual image 302 would appear to be out of focus (e.g.,blurry) at the second time t₂.

FIG. 4 illustrates a method 400 of operation of the near-eye displaysystem 100 for rendering lightfield frames using lenslets withadjustable focal lengths to provide dynamic image focus adjustments inaccordance with some embodiments. The method 400 illustrates oneiteration of the process for rendering and displaying a lightfield framefor one of the left-eye display 110 or right-eye display 112, and thusthe illustrated process is repeatedly performed in parallel for each ofthe displays 110, 112 to generate and display a different stream orsequence of lightfield frames for each eye at different points in time,and thus provide a 3D, autostereoscopic VR or AR experience to the user.

For a lightfield frame to be generated and displayed, method 400 startsat block 402, whereby the rendering component 402 identifies the imagecontent to be displayed to the corresponding eye of the user as alightfield frame. In at least one embodiment, the rendering component104 receives the IMU information 152 representing data from variouspose-related sensors, such as a gyroscope, accelerometer, magnetometer,Global Positioning System (GPS) sensor, and the like, and from the IMUinformation 150 determines a current pose of the apparatus 114 (e.g.,HMD) used to mount the displays 110, 112 near the user's eyes. From thiscurrent pose, the CPU 136, executing the rendering program 144, candetermine a corresponding current viewpoint of the subject scene orobject, and from this viewpoint and graphical and spatial descriptionsof the scene or object provided as rendering information 148, determinethe imagery to be rendered for the current pose.

At block 404, the CPU 136, executing eye tracking program 146,determines the current pose of the corresponding eye of the user. Asexplained herein, the current pose of an eye may be determined using anyof a variety of eye tracking techniques. Generally, such techniquesinclude the capture of one or more images of IR light reflected from thepupil and cornea of the eye. The eye tracking program 146 then maymanipulate the CPU 136 or the GPUs 138, 140 to analyze the images todetermine the pose of the eye based on the corresponding position of oneor both of the pupil reflection or corneal reflection. Further, theorientation of the pupil relative to the cornea in turn may be used todetermine the orientation of the eye (that is, the direction of gaze ofthe eye). It should be noted that although block 404 is illustrated inFIG. 4 as being subsequent to block 404, the process of block 404 may beperformed before, during, or after the process of block 402.

With the current pose of the user's eye determined, at block 406 therendering program 144 manipulates the CPU 136 to calculate a desiredfocal length (e.g., to desired a focal point or a focal plane) for oneor more lenslets 126 in lenslet array 124, based on the current pose ofthe user's eye. As explained above, the focal length represents thedistance at which the sharpest focus is attained when viewing light(carrying image data) projected from the lenslet. In particular, thedesired focal length is intended to allow for image elements (e.g.,objects, individuals, scenes and the like in the virtual image) that theuser's eye gaze is directed at to be perceived as being in focus afterprojecting through the lenslet 126. That is, the desired focal lengthserves to dynamically adjust the distance at which light rays projectedfrom the lenslets converge to match the current pose of the eye, therebychanging the focus at which different views of the image content areperceived.

In at least one embodiment, the calculation of the desired focal lengthis based at least in part on an identification of a virtual object inthe virtual image that the user is directing his or her gaze towardsusing the current pose of the user's eye relative to the display panel118. To illustrate, referring to an example scenario illustrated bycross-section view 500 of FIG. 5, the virtual image can include a numberof objects intended to be perceived by the eye 132 at different depths.For example, the virtual image includes image data representing acoconut 502 and a tree 504 positioned at depths d₁ and d₂ within thevirtual image, respectively. Assuming in this example that the currentpose of the eye as determined in block 404 determines that the user'sgaze is focusing on the coconut 502, the desired focal length may becalculated such that light containing image data for the coconut 502projected from lenslet 126-3 is focused at a first focal point 508 atthe back of the eye 132. Accordingly, the coconut 502 in the virtualimage would appear to be in focus.

In some embodiments, the calculation of the desired focal length furtherincludes the determination of an accommodation range, which generallyrefers to a range of depths at which objects in the virtual image willbe perceived as in focus. Objects positioned within the accommodationrange may be perceived to be in focus; objects positioned outside theaccommodation range (i.e., at a virtual depth too close or too far awayfrom the eye) will not be perceived to be in focus even if the currentpose of the eye is gazing directly at that out of accommodation rangeobject. For example, referring again to FIG. 5, the tree 504 would notbe perceived as in focus even if the current pose of the eye is gazingdirectly at the tree 504 as it is positioned outside the accommodationrange 506.

In contrast, if the accommodation range 512 had been determined, boththe coconut 502 and tree 504 could appear to be in focus to the eye 132.In particular, assuming in this example that the current pose of the eyeas determined in block 404 determines that the user's gaze is focusingon the coconut 502 at a first time t₁, the coconut 502 in the virtualimage would appear to be in focus as it is positioned within theaccommodation range 512. In some embodiments, at the first time t₁, thetree 504 in the virtual image is positioned within the accommodationrange 512 but is not perceived to be in focus as the current pose of theeye is focused on the coconut 502. That is, the determination of theaccommodation range further includes determining one or more desiredfocal lengths such that objects positioned within the accommodationrange, but are not focused on by the user's gaze, are not perceived tobe completely in focus.

The focal lengths can be determined to provide one or more of focusareas in which directed gaze perceives objects in focus and defocusareas in which defocus blur is provided, thereby providing accommodationcues in the form of retinal blur to aid in the simulation of depthperception. However, at a subsequent time t₂, if the user's gaze ischanged to focus on the tree 504, the tree 504 in the virtual imagewould appear to be in focus as it is positioned within the accommodationrange 512. Similarly, the coconut 502 is positioned within theaccommodation range 512 but is not perceived to be in focus as thecurrent pose of the eye at the second time t₂ is focused on the tree504.

In other embodiments, the calculating of a desired focal length in block406 can optionally include a compensation for existing refractive errorsin the user's eye (e.g., myopia, hyperopia). For example, a flat shiftmay be applied to the desired focal distance for each portion of theintegral lightfield to correct for nearsightedness or farsightedness ofthe user, enabling the image to be viewed in focus by a user whonormally must wear corrective lenses (e.g., glasses or contact lenses)without wearing such corrective lenses. Similar compensations can alsobe applied to account for mechanical/thermal drift due to environmentalconditions or assembly tolerances from manufacturing.

With the desired focal length determined, at block 408 the renderingprogram 144 manipulates the CPU 136 to calculate a voltage to be appliedto a variable-index material. As part of this process, the CPU 136 alsoinstructs the calculated voltage to be applied for inducing a change inthe refractive index of the variable-index material, which in turncauses a change in the incidence angles of light entering and exitingthe lenslets discussed herein. For example, referring back to FIGS. 1and 2, some embodiment include constructing the lenslets out of thevariable-index material. Accordingly, applying the calculated voltage tothe lenslets directly changes their refractive indexes and incidenceangles of light entering and exiting the lenslet array. In otherembodiments, such as discussed relative to FIG. 3, the variable-indexmaterial can be provided as a layer disposed between the display panel118 and the lenslet array 124. In such embodiments, applying thecalculated voltage to the layer of variable-index material 158 onlydirectly changes the refractive index and incidence angles of lightentering and exiting the layer of variable-index material 158. However,the change to incidence angles of light entering and exiting the layerof variable-index material 158 results in a change to the incidenceangles of light received by the lenslet array 124, thereby changing thefocus point and length of the lenslets 126. The GPU subsequently rendersthe lightfield frame at block 210 and provides the lightfield frame tothe corresponding one of the computational displays 110, 112 for displayto the eye 132 of the user with the adjustment to focal lengths ofblocks 406 and 408. It should also be noted that although block 410 isillustrated in FIG. 4 as being the last step of method 400, the processof block 410 may also be performed before, during, or after the processof block 402.

As explained above, the dynamic accommodation range adjustment and focallength change process described herein utilizes an eye trackingcomponent (e.g., eye tracking components 106, 108) to determine thecurrent pose of a corresponding eye. This eye tracking componenttypically includes one or more IR illuminators to illuminate the eye, animaging camera to capture imagery of IR reflections from the eye, one ormore lenses, waveguides, or other optical elements to guide thereflected IR light from the eye to the imaging camera, and one or moreprocessors executing a software program to analyze the captured imagery.

FIG. 5 illustrates an additional example computational display such asthe ones utilized in the near-eye display system 100 for accommodationrange extension using variable-index materials in accordance with someembodiments. As shown by the cross-section view 500, in thisconfiguration, each of the lenslets 126 of the lenslet array 124 servesas a separate “projector” onto the eye 132, with each “projector”overlapping with one or more adjacent projectors in forming a compositevirtual image from the array of elemental images displayed at thedisplay panel 118.

As shown by view 500, a virtual image can include a number of objectsintended to be perceived by the eye 132 at different depths. Forexample, the virtual image includes image data representing a coconut502 and a tree 504 positioned at depths d₁ and d₂ within the virtualimage, respectively. Assuming in this example that a current pose of theeye 132 determines that the user's gaze is focusing on the coconut 502at a first time t₁, the refractive index of the layer of variable-indexmaterial 158 may be computed (e.g., by rendering component 104) andelectrically changed such that light containing image data as projectedfrom the lenslets is associated with an accommodation range 506.

Image data for the coconut 502 projected from lenslet 126-3 is focusedat a first focal point 508 at the back of the eye 132. Accordingly, thecoconut 502 in the virtual image would appear to be in focus at thefirst time t₁. However, based on the refractive index of the layer ofvariable-index material 158 at the first time t₁, light containing imagedata for the tree 504 from lenslet 126-1 is focused at a second focalpoint 510. In other words, the tree 504 is positioned outside of theaccommodation range 506. Accordingly, the tree 504 in the virtual imagewould appear to be out of focus (e.g., blurry) at the first time t₁. Incontrast, if the refractive index of the layer of variable-indexmaterial 158 had been computed to generate an accommodation range 512,both the coconut 502 and tree 504 would appear to be in focus to the eye132.

FIG. 6 is a diagram illustrating an example varifocal lenslet array fordynamic focal length adjustment in the near-eye display system of FIG. 1in accordance with some embodiments. As shown by the top view 600, inthis configuration, the lenslet array 124 includes a first array 602 ofcubic phase plates 604 and a second array 606 of cubic phase plates 604.Spatially translating the first array of cubic phase plates 602 relativeto the second array of cubic phase plates 604, such as by the lateraldisplacement between the two arrays as illustrated in FIG. 6, changesthe focal length of the cubic phase plates 604. By translating twosuperimposed cubic phase functions, a variable quadratic (i.e.,varifocal) effect is introduced. Similarly, the lenslet array 124 caninclude two arrays of freeform phase plates, such as Lohmann-Alvarezvarifocal lenses in which the focal length of the lenses is changed by alateral displacement between the lenses. This enables dynamic focallength adjustment by using well defined surface functions. In someembodiments, certain aspects of the techniques described above mayimplemented by one or more processors of a processing system executingsoftware. The software comprises one or more sets of executableinstructions stored or otherwise tangibly embodied on a non-transitorycomputer readable storage medium. The software can include theinstructions and certain data that, when executed by the one or moreprocessors, manipulate the one or more processors to perform one or moreaspects of the techniques described above. The non-transitory computerreadable storage medium can include, for example, a magnetic or opticaldisk storage device, solid state storage devices such as Flash memory, acache, random access memory (RAM) or other non-volatile memory device ordevices, and the like. The executable instructions stored on thenon-transitory computer readable storage medium may be in source code,assembly language code, object code, or other instruction format that isinterpreted or otherwise executable by one or more processors.

A computer readable storage medium may include any storage medium, orcombination of storage media, accessible by a computer system during useto provide instructions and/or data to the computer system. Such storagemedia can include, but is not limited to, optical media (e.g., compactdisc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media(e.g., floppy disc, magnetic tape, or magnetic hard drive), volatilememory (e.g., random access memory (RAM) or cache), non-volatile memory(e.g., read-only memory (ROM) or Flash memory), ormicroelectromechanical systems (MEMS)-based storage media. The computerreadable storage medium may be embedded in the computing system (e.g.,system RAM or ROM), fixedly attached to the computing system (e.g., amagnetic hard drive), removably attached to the computing system (e.g.,an optical disc or Universal Serial Bus (USB)-based Flash memory), orcoupled to the computer system via a wired or wireless network (e.g.,network accessible storage (NAS)).

Note that not all of the activities or elements described above in thegeneral description are required, that a portion of a specific activityor device may not be required, and that one or more further activitiesmay be performed, or elements included, in addition to those described.Still further, the order in which activities are listed are notnecessarily the order in which they are performed. Also, the conceptshave been described with reference to specific embodiments. However, oneof ordinary skill in the art appreciates that various modifications andchanges can be made without departing from the scope of the presentdisclosure as set forth in the claims below. Accordingly, thespecification and figures are to be regarded in an illustrative ratherthan a restrictive sense, and all such modifications are intended to beincluded within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any feature(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature of any or all the claims. Moreover, the particular embodimentsdisclosed above are illustrative only, as the disclosed subject mattermay be modified and practiced in different but equivalent mannersapparent to those skilled in the art having the benefit of the teachingsherein. No limitations are intended to the details of construction ordesign herein shown, other than as described in the claims below. It istherefore evident that the particular embodiments disclosed above may bealtered or modified and all such variations are considered within thescope of the disclosed subject matter. Accordingly, the protectionsought herein is as set forth in the claims below.

What is claimed is:
 1. In a near-eye display system, a methodcomprising: determining, using an eye tracking component of the near-eyedisplay system, a first pose of a user's eye; determining a desiredfocal point for an array of elemental images forming an integrallightfield frame based on the first pose of the user's eye byidentifying a virtual depth of a target object focused on by the firstpose of the user's eye relative to the virtual depths of one or moreother objects within the integral lightfield frame; changing the focallength of light projecting out of a lenslet array based on the firstpose of the user's eye and the desired focal point; rendering the arrayof elemental images at a position within the integral lightfield framebased on the changed focal length of light; and displaying the integrallightfield frame at a display panel of the near-eye display system. 2.The method of claim 1, wherein determining the first pose of the user'seye comprises: capturing imagery of the user's eye using an imagingcamera disposed between the display panel and the lenslet arrayoverlying the display panel.
 3. The method of claim 1, wherein changingthe focal length comprises: applying a voltage to a variable-indexmaterial disposed between the display panel and the lenslet array toinduce a change in a refractive index of the variable-index material,wherein the change in the refractive index causes a change in theincidence angles of light entering and exiting the lenslet array.
 4. Themethod of claim 1, wherein changing the focal length comprises: applyinga voltage to one or more lenslets in the lenslet array comprising avariable-index material to induce a change in a refractive index of theone or more lenslets, wherein the change in the refractive index causesa change in the incidence angles of light entering and exiting thelenslet array.
 5. The method of claim 1, wherein changing the focallength comprises: applying a voltage to a variable-index materialassociated with a first portion of the lenslet array to induce a changein a refractive index of the variable-index material, wherein the firstportion of the lenslet array is independently addressable relative to asecond portion of the lenslet array, and further wherein the change inthe refractive index causes a change in the incidence angles of lightentering and exiting the first portion without changing the incidenceangles of light entering and exiting the second portion of the lensletarray.
 6. The method of claim 1, wherein determining the desired focalpoint for the array of elemental images further comprises: determiningthe desired focal point to compensate for existing refractive errors inthe user's eye.